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ABSTRACT 


This  discussion  of  the  present  state  of  systems  analysis  was  delivered  as 
a  talk  at  the  20th  Military  Operations  Research  Symposium,  held  12  December 
196?,  et  the  National  Bureau  of  Standards,  Gaithersburg,  Maryland. 

This  paper  published  as  a  Research  Contribution  to  provide  as  early  as 
possible  the  results  of  CNA  supporting  analysis  to  the  Naval,  Marine,  and  ana¬ 
lytical  community. 
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WHERE  IS  SYSTEMS  ANALYSIS? 


Most  practioners  of  the  art  of  systems  analysis  seem  to  agree  that  it  has 
suece  *ded  well  in  some  areas.  In  others,  they  see  hopeful  signs,  in  still 
others,  the  feeling  is  that  a  long  time  may  pass  before  real  progress  is  made. 

It  is  in  keeping  with  the  traditions  of  this  art  to  explore  critically  and  try  to 
understand  why  there  is  this  significant  consensus  among  practitioners  about 
the  usefulness  of  systems  analysis. 

Let  us  consider  the  framework  of  systems  analyses.  First,  these  analyses 
are.  almost  exclusively,  "force -package" -related.  It  is  no  coincidence  that 
program  budgeting  and  system  analysis  have  succeeded  together.  The  existence 
of  a  specific  set  of  programs,  with  associated  costs  and  outputs  has  made  it 
possible  -  or.  at  least,  easier  -  to  focus  the  systems  analysis  effort. 

At  the  same  time,  however,  analysts  have  become,  to  some  extent  prisoners 
of  the  categories  of  output.  For  example,  though  strategic  mobility  forces  are 
farther  than  strategic  nuclear  forces  from  final  output,  analysts  treat  both  in 
much  the  same  way.  Few  studies  of  general-purpose  forces  treat  rigorously  the 
implications  of  alternative  systems  for  strategic  mobility  forces. 

Little  systematic  analysis  has  been  conducted  on  the  choices  among  major 
force  packages  or,  in  the  case  of  general-purpose  forces,  among  their  separate 
components.  Nor  has  much  study  been  devoted  to  such  obvious  questions  as  the 
relationship  between  activ  ad  reserve  forces  of  a  given  type.  Perhaps  the 
areas  of  analytical  achievement  would  be  different  if  the  forces  were  packaged 
differently.  Perhaps  more  would  be  known  about  the  air/ground  trade-off  if 
tactical  air  and  ground  forces  were  not  packaged  separately. 

In  addition  to  being  related  to  force  package  problems,  most  systems  analyses 
are  assessed  m  terms  of  the  ability  to  answer  adequately  a  hierarchy  of  questions 
of  practical  interest  to  decision-makers  at  the  level  of  the  Department  of  Defense 
and  the  services.  In  descending  order,  they  are 

First,  questions  about  the  level  of  force  in  a  package; 

Second,  questions  about  the  mix  of  land-  or  sea -based  systems  for  a  given 
level  of  capability  or  budget.  This  is  a  special  set  which  is  logically  little  dif¬ 
ferent  from  the  third  set  of  questions,  but  the  obvious  service  interests  give  it 
special  emphasis.  Not  surprisingly,  we  have  concentrated  a  good  part  of  our 
efforts  on  trade-offs  between  Polaris  and  Minuteman.  carriers  and  land-based 
air.  and  C-5  aircraft  and  FDL  ships. 

Third,  there  are  questions  abet  the  choice  for  procurement  of  .-.jx'cific 
systems  or  subsystems,  such  as  an  aircraft,  tank,  ship,  or  rifle  type.  Notable 
examples  are  the  .-111.  the  main  battle  tank,  nuclear  powered  ships,  and  the 
M-lh  rifle. 
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The  concern  with  these  large  and  immedj  .te  procurement  and  force-size 
questions  has  relegated  a  fourth  category  to  relative  obscurity.  These  are 
questions  about  "R&D  strategies"  on  which  much  work  used  to  be  done.  How 
much  should  we  be  willing  to  pay  to  find  out  more  about  alternative  systems 
before  we  make  system  choices?  How  many  systems  should  we  simultaneously 
develop  in  the  face  of  uncertainty  about  their  relative  costs  and  productivity  ? 

In  other  wo ’•'is,  hmv  big  a  menu  of  alternatives  should  wc  preserve  for  the 
future,  and  how  long  should  we  delay  to  gain  information  before  making  decisions? 
The  problems  arc  tractable.  There  is  at  least  a  partial  body  of  theory  to  work 
from.  Moreover,  good  solutions  very  probably  have  greater  leverage  to  help 
the  American  taxpayer  in  the  long  run  than  the  questions  we  have  concentrated 
on.  Finally  such  analyses  should  help  to  reduce  part  of  the  multi-faceted  un¬ 
certainty  that  confounds  us  in  dealing  with  certain  force  packages. 

We  shall  return  to  this  point,  but  first,  what  is  the  assessment  of  systems 
analysis?  And  why?  There  seems  to  be  general  agreement  that  in  the  strategic 
nuclear  offensive  and  defensive  force  packages  we  have  things  fairly  well  in 
hand;  we  contribute  significantly  to  questions  of  level,  mix,  and  subsystem 
choice.  In  the  area  of  strategic  mobility  we  also  feel  comfortable  about  making 
recommendations  not  only  about  the  mix  of  systems  but  the  level  of  capability 
the  U.  S.  should  procure.  But  this  widely  accepted  favorable  assessment  of  the 
analvfif'al  contribution  to  these  programs  descends  when  we  begin  to  consider 
tactical  air  and  ASW.  And  finally,  there  are  very  few  positive  statements  about 
what  systems  analysis  has  done  to  clarify  decisions  on  questions  about  conventional 
ground  forces.  In  this  area  wc  can  barely  make  PERT  charts  for  procurements 
associated  with  force  buildups. 

The  same  ordering  of  the  conti  buttons  of  systems  analysis  is  present  in  the 
evidenced  attitude  of  decision-makers  toward  each  force  package. 

in  the  strategic  nuclear  and  strategic  mobility  areas,  decision-makers  are 
both  able  and  willing  to  make  very  explicit  judgments  about  the  objective  or 
criteria  in  terms  which  are  susceptible  of  calculation,  given  the  state  of  the  art 
of  systems  analysis.  For  example,  with  respect  to  strategic  forces,  decision¬ 
makers  are  willing  to  say  more  than  merely  that  we  should  be  able  deliver  a 
second  strike  attack  of  sufficient  magnitude  to  deter  the  enemy  from  a  first 
strike:  They  are  actually  willing  to  spell  out  what  that  level  is  in  terms  of  the 
percentage'  of  enemy  population  and  industrial  capacity  destroyed.  While  these 
assessments  of  the  specific  level  of  destruction  needed  for  deterrence  hove 
changed  over  time,  this  has  not  shaken  the  decision-makers  faith,  or  ours,  in 
their  ability  to  make  the  appropriate  explicit  judgments  and  to  use  analysis  in 
arriving  at  those  judgments.  The  important  point  is  that  where  the  decision¬ 
maker  is  willing  to  stick  his  neck  out.  the  analyst's  problem  is  radically  simplified. 

In  the  case  of  rapid  deployment  forces,  decision-makers  arc  willing  to  spell 
out  in  some  detail  the  specific  time  profiles  of  delivery  of  forces  to  particular 
geographic  areas  and  to  talk  about  how  many  simultaneous  contingencies  we  should 
be  able  to  handle.  Decision-makers  do  this  in  spite  of  the  fact  that  our  knowledge 
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of  conventional  warfare  is  crude.  We  have  a  poor  understanding  of  how  much  it 
is  worth  to  deliver  a  division  in,  say  7  days  instead  of  4.  Consequently,  speci¬ 
fications  of  delivery  profiles  can  only  be  based  upon  the  roughest  of  judgments. 

Yet,  once  given,  they  do  make  detailed  calculation  possible. 

'n  going  through  the  other  force  packages  -  those  packages  in  which  the 
assessment  of  the  contribution  of  systems  analysis  is  lower  -  the  character  of 
the  judgment  decision-makers  can  or  are  willing  to  make  is  different.  For 
tactical  air  foxces,  for  example,  there  are  few  useable  objectives  or  criteria, 
other  than  very  crude  ones,  for  determining  how  much  force  we  should  have. 

This  is  also  true  about  land  forces.  The  conceptual  problem  of  assigning  explicit 
objectives  or  criteria  in  these  areas  is  no  different,  in  a  broad  sense,  than  it  is 
for  strategic  nuclear  forces.  Yet  they  are  not  assigned. 

The  actions  of  decision-makers  shouldn't  surprise  us.  The  analysis  of 
strategic  forces  is  at  least  sufficient  to  indicate  how  changing  expenditures  on 
weapons  will  affect  the  percentage  of  surviving  enemy  populations  or  industrial 
capacities.  For  general -purpose  forces,  we  poorly  understand  what  desirable 
outcomes  are  or  how  changes  in  the  forces  affect  outcomes.  In  these  areas, 
the  primary  form  of  analysis  used  in  force  level  decisions  is  a  comparison  of 
friendly  and  enemy  forces  on  several  different  standards.  For  example,  in  an 
analysis  of  tactical  aviation  we  depend  on  such  comparisons  as  the  total  payload 
capability  of  the  two  opposing  total  tacical  air  forces  computed  for  a  single 
sortie  on  a  mission  of  fixed  radius.  For  ground  forces,  the  comparisons  are 
closer  to  simple  personnel  or  equipment  counts,  with  some  account  taken  of 
qualitative  differences. 

It  is  perfectly  reasonable  to  ask.  What  is  so  peculiar  about  the  areas  v  here 
we  believe  systems  analysis  has  made  a  contribution?  Why  are  they  arrayed 
this  way0 

In  trying  to  address  these  questions,  and  at  the  risk  of  appearing  facetious, 
it  may  be  appropriate  to  relate  one  of  Parkinson's  Laws.  Parkinson  observed 
that  in  the  boards  of  large  corporations  multi-million  dollar  decisions  were 
made  rather  quickly  while  decisions  about  such  trivial  matters  as  putting  a  roof 
over  the  employees'  bike  shed  provoked  long  and  spirited  debate.  This  is  an 
example  of  the  "Law  of  Triviality.  "  which  states  that  the  time  spent  on  any  item 
on  the-  agenda  will  be  in  inverse  proportion  to  the  sum  involved.  The  explanation 
is  that  events  and  magnitudes  bevond  the  realm  of  personal  experience  may  seem 
considerably  less  complex  than  those  which  can  be  related  to  individual  experience. 

Tins  human  trait  does  not  appear  to  provide  a  complete  explanation  of  the 
differential  attitude  to  problems  in  the  various  "force-program"  areas,  but 
there  may  be  a  grain  of  truth  in  it.  Current  opinion  as  to  the  success  of  systems 
analysis  is  very  negatively  correlated  with  the  amount  of  human  experience  in  each 
area.  For  example,  there  has  been  no  experience  with  massive  nuclear  exchanges 
and  very  little  with  rapid  deployment,  but  we  believe  systems  analysis  has  led  us 
to  understand  these  programs.  It  would  appear  that  opinion  is  split  on  how  well 
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we  do  now  or  will  do  in  the  immediate  future  with  the  analysis  of  tactical  air 
forces;  we’ve  had  a  little  experience  the:‘e,  The  same  applies  to  ASW  forces. 

Finally,  at  the  other  pole,  we  find  ground  combat.  The  verdict  of  experts 
is  that  it  is  nearly  intractable.  Experience  there,  in  one  form  or  another,  has 
been  piling  up  since  the  African  genesis.  On  its  face,  this  is  a  strange  correla¬ 
tion.  Most  sciences  are  based  on  observation  and  are  designed  to  permit  us  to 
extend  experience. 

But  then,  this  explanation  is  based  upon  simple  correlation.  Although  it  is 
an  extremely  close  one,  there  is  always  the  possibility  that  it  is  spurious  and 
that  the  contribution  of  systems  analysis  to  the  understanding  of  each  force 
package  is  really  explained  by  cau  al  factors  other  than  lack  of  evidence. 

For  example,  some  areas  may  be  neater  or  inherently  more  tractable  than 
others.  The  more  the  subjects  of  investigation  are  pre-planned  and  the  more 
they  depend  on  the  characteristics  of  large  pieces  of  equipment  with  few  operating 
modes,  and  the  more  they  are  determined  by  early  large  events,  and  the  more 
likely  they  are  to  be  analytically  tractable.  These  areas  are  less  likely  to  be 
influenced  by  unpredictable  behavior  and  states  of  training  or  to  be  dependent 
upon  a  myriad  of  small  encounters.  Our  assessment  of  those  characteristics 
in  each  of  the  force  packages  also  tends  to  correlate  nicelv  with  how  we  feel 
about  the  state  of  analysis  in  each. 

There  may  even  be  a  more  simple  explanation.  Given  the  present  state  of 
the  art  of  analysis,  we  can  adequately  analyze  conflicts  which  happen  with  such 
great  speed  that  there  is  no  real  opportunity  to  re-assess,  re-deploy  forces  or 
change  tactics.  Conversely,  the  analysis  of  ionger  conflicts  in  which  each  side 
has  the  opportunity  to  learn  from,  and  react  to,  the  experience  of  the  war  cannot 
be  convincingly  done  tod./. 

When  one  considers  the  most  widely  used  analyses  of  conventional  wars,  a 
remarkable  number  are  high  intensity,  extremely  short  wars.  For  example, 
most  of  the  analyses  which  deal  with  a  NATO  war  look  much  more  like  the  recent 
Arab-lsrali  conflict  than  World  War  II  revisited.  All  decisive  action  is  taken 
and  the  outcome  is  decided  in  about  one  week  Similarly,  in  ASW.  the  w-ars 
analyzed  are  incredibly  quick.  But  the  users  of  the  analysis  believe  that  neither 
we  nor  the  enemy  would  act  in  the  specific  ways  suggested.  As  th-  length  of  the 
conflict  grows,  this  objection  gains  much  more  force.  Both  we  and  our  enemies 
would  continually  re -adapt  our  tactics  and  strategies  to  what  is  happening,  and 
the  problem  of  analysis  becomes  much  more  complex.  It  is  not  surprising  that 
the  analysis  of  "broken -back"  nuclear  exchanges  is  less  advanced  than  that  of 
massive  exchanges.  The  difference  between  strategic  nuclear  and  strategic 
conventional  wars  is  that  in  the  latter  case  the  massive,  rapid,  single-strategy 
conflict  is  considered  to  be  a  good  deal  less  likely.  In  any  case,  it  is  a  good 
deal  less  convincing. 
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There  is  another  explanation  which  is,  in  part,  related  to  the  correlation 
between  experience  with  a  type  of  force  and  the  ability  to  analyze  it.  That  is, 
the  people  who  do  systems  analysis  probably  have  taken  the  trouble  to  learn  a 
great  deal  more  about  the  details  of  strategic  nuclear  forces  and  are  as  expert 
about  the  fundamentals  as  any  other  group.  This  may  be  because  there  is  little 
opportunity  for  field  experimentation,  and  what  little  is  done  is  carefully  cata¬ 
logued.  As  a  consequence,  it  is  amenable  to  a  form  of  research  that  anu’ysts 
are  more  accustomed  to.  With  conventional  forces,  systems  analysts  are  prob¬ 
ably  a  good  deal  less  expert  than  the  operators.  Given  the  competitive  environ¬ 
ment  in  which  most  analysis  is  necessarily  done,  this  lack  of  expertise  is  a 
weakness  that  is  likely  to  be  fatal. 

Of  course,  there  has  been  more  opportunity  for  experience  with  general - 
purpose  forces  to  disprove  an  analysis.  The  number  of  carefully  constructed 
and  executed  tests  of  weapons  systems  in  appropriate  environmental  conditions 
is  and  has  been  very  small.  As  a  result,  there  can  be  little  doubt  that  our 
peacetime  estimates  of  the  wartime  performance  of  systems  are  very  poor.  Be¬ 
fore  the  Vietnamese  conflict  we  badly  misestimated  our  capability  with  certain 
weapon  systems.  Even  for  certain  narrowly  defined  tactical  air  missions,  such 
as  the  number  of  sorties  to  achieve  a  kill  on  a  specified  target  with  a  given  weap¬ 
on,  we  found  some  estimates  off  by  an  order  of  magnitude. 

Hus  is  nothing  new.  although  we've  improved  somewhat  Herman  Goering 
claimed  that  no  enemy  ixunber  would  ever  attack  Berlin.  His  estimate  of  the 
number  of  anti-aircraft  shells  required  per  aircraft  kill  was  off  by  a  factor  of 
1.000.  file  guns  used  in  the  German  tests  were  manned  by  men  who  might  be1 
characterized  as  athletes  with  PhD’s  in  physics. 

It  doesn't  take  a  war  to  demonstrate  a  misestimate  of  performance.  Systems 
analyses  are  generally  directed  at  the  procurement  of  new,  or  relatively  new. 
systems.  We  tend  to  accept  the  estimates  of  the  potential  producer,  who  has  a 
strong  incentive  to  make  the  system  look  good.  The  alternative  source  of  data 
is  the  "in-house"  project  manager,  who  is  also  not  likely  to  ho  very  ill-disposed 
to  the  system  e'-  n  if  he  has  any  real  uncertainty  about  it. 

When  the  system  actually  gets  into  use  the  analyst  is  often,  for  i  number 
of  reasons,  the  last  person  to  find  out  how  well  or  badly  it  performs.  Given  the 
way  we  operate  now.  there  is  often  a  significant  delay  before  the  analysts  know 
what’s  happening  in  operational  exercises.  Ibis  is  more  evident  for  conventional 
general -purjiosc  forces,  because  more  widespread  and  un reported  training  and 
practice  occurs  with  them,  and  the  analysts  arc  not  present. 

1  (vrsonally  lxlieve  that  the  difficulty  experienced  in  dealing  with  conventional 
general-purpose  forces  is  a  composite  of  these  factors.  Ibc  situations  arc  signif¬ 
icantly  more  complex,  tactics  and  strategy  changes  brought  about  by  derived  in¬ 
formation  during  the  conflict  must  lx-  addressed;  and  finally,  systems  analysts 
are  relatively  inexpert  ir  these  areas  .end  insufficiently  oriented  empirically. 


As  a  result,  there  are  large  and  recognized  uncertainties  operating  in 
several  dimensions  which  drive  through  many  of  the  large  general-purpose 
force  problems  we  ai'empt  to  deal  with.  If  we  explicitly  recognize  each  un¬ 
certainty  and  try  to  do  sensitivity  analyses,  the  value  of  the  analysis  as  a  pre¬ 
dictor  of  outcome  may  not  be  meaningless,  but  as  an  aid  to  decisicn-malcing, 
its  utility  is  very  low.  It  has  been  observed  in  these  studies  that  there  are 
reasonable  assumptions  which  provide  results  of  totally  different  character, 
not  only  as  to  outcome  of  war  situations,  but  also  as  to  the  relative  attractive¬ 
ness  of  different  systems  tested.  What  we  often  end  up  with  is  either  a  myriad 
of  possible  outcomes,  with  little  ability  to  select  the  most  reasonable  ones,  or 
with  a  few  point  estimates  that  we  must  qualify  so  broadly  that  they  lack  credence. 
It's  not  surprising  that  when  it  comes  to  general-purpose  forces  the  decision¬ 
makers  are  unwilling,  or  more  accurately,  unable  to  state  a  specific  objective, 
as  they  do  for  strategic  nuclear  forces.  They  can  state  only  the  most  general 
rules  for  choice. 

Some  of  the  uncertainty  results  from  incompletely  known  technical  performance 
some  reflects  our  ignorance  of  how  systems  interact  in  conflicts,  and  some  is  a 
result  of  uncertainty  about  how  information  derived  from  the  conflict,  affects  the 
contestants'  actions. 

These  are  new  techniques  which  would  permit  us  to  deal  with  this  third  kind 
of  uncertainty  by  radically  expandim  our  p'  ility  to  game  problems  while  avoiding 
the  previous  limitations  of  analytic  content.  This  is  very  interesting.  However, 
such  a  method  is  subject  to  the  same  explosion  of  results  and  effort,  and  most 
of  the  necessary  data  base  and  modularized  analytical  models  are  themselves 
subject  to  wide  variations  of  the  first  two  kinds,  ia  other  words,  we  now  have 
problems  of  great  uncertainty  in  dealing  wim  v  fixed  single,  preplanned  decision 
process. 

If  this  is  all  tiue,  then  our  primary  task  may  be  to  fi  1  some  way  of  reducing 
some  of  the  uncertainty.  The  well-known  task  for  the  sensitivity  test  is  to  help 
find  out  what  uncertainties  matter  most.  The  next  step  is  to  determine  which  of 
those  uncertainties  that  matter  are  ones  we  can  really  hope  to  narrow  -  and  then 
do  something  about  them. 

It  is  the  latter  task  -  of  doing  so  nothing  about  the  uncertainties  that  matter  - 
which  we  have  tended  to  ignore.  It  is  probably  because  systems  analysts  like  to 
think  of  themselves  much  more  as  "conccptualizers”  rather  than  as  empiricists. 

For  example,  how  can  anyone  have  much  confidence  in  the  validity  of  a  model 
for  predicting  certain  outcomes  of  tactical  air  encounters  while  simultaneously 
adn  itting  that  this  knowledge  of  the  inputs  is  poor?  The  lack  of  agreement  about 
inputs  may  be  no  fault  of  the  model,  but  it  raises  questions  as  to  its  validity. 

Models  are  abstract  representations  of  reality  which  help  us  to  perceive 
significant  relationships  in  the  real  world,  to  manipulate  them,  and  thereby  pre¬ 
dict  outcomes.  The  i  cal  test  of  their  utility  or  validity  is  whether  H-ey  are  good 
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predictors.  With  the  admittedly  great  uncertainty  about  tactical  and  technical 
inputs  or  outcomes,  it  is  essentially  impossible  to  determine  the  validity  of  a 
model.  We  simply  can't  make  that  judgment  now. 

Systems  analysis  does  need  more  empiricists.  We  need  more  people  who 
are  involved,  not  only  in  structuring  complex  problems,  but  who  bring  this 
understanding  to  the  development  and  monitoring  of  tests.  Further,  the  tests 
must  be  conducted  in  an  organizational  framework  which  at  least  eliminates  the 
most  obvious  bias -forming  incentives.  Moreover,  the  test  should  be  conducted 
under  a  spectrum  of  operating  conditions,  not  "ideal"  conditions  alone.  The 
single  most  important  need  for  the  future  success  of  systems  analysis  in  dealing 
with  general-purpose  force  issues  is  the  acquisition  of  good  empirical  data.  It 
is  a  task  we  cannot  leave  to  others. 

The  responsibility  for  the  inadequacies  of  the  present  state  of  knowledge  can 
hardly  be  laid  exclusively  on  the  community  of  systems  analysts.  There  has  been 
a  real  conflict  between  the  use  of  operational  units  in  training  exercises  and  their 
use  in  monitored  testing.  Commanders  of  operational  forces  have  tended  to  pre¬ 
fer  training  to  testing,  a  preference  which  may  well  be  shortsighted. 

Finally,  we  turn  to  a  largely  ignored  but  important  task  mentioned  earlier  - 
work  on  R  and  D  strategies.  This  is  a  special  case  of  the  general  problem  of 
dealing  with  uncertainty.  It  is  generally  recognized  that  our  predictions  of  the 
costs,  time  to  develop,  and  success  of  developments  of  new  weapons  systems 
have  been  poor.  Cost  increases  of  200  to  300  percent  and  extensions  of  develop¬ 
ment  time  by  30  to  50  percent  are  the  rule.  Moreover,  the  size  of  the  error  has 
been  very  variable. 

Recently,  CNA  reviewed  some  predictions  made  in  1958  by  the  best  available 
experts  about  ASW  system  performances  in  1965.  The  .1965  actual  data  indicated 
wide  errors.  For  example,  the  estimate  of  the  ability  of  a  CODAR  pair  to  localize 
a  conventional  submarine  target  was  off  by  a  factor  of  1C.  Only  one  system  was 
projected  with  reasonable  accuracy. 

The  purpose  of  the  estimates  of  cost  and  performance  is  to  permit  better 
decisions.  Given  the  history  of  the  accuracy  of  these  estimates,  we  may  be 
failing  in  this  task.  There  is  undeniable  optimistic  bias,  but  the  effect  is  rh^r 
the  systems  where  we  make  the  woist  errors  tend  to  be  the  ones  we  favor.  That 
is,  the  systems  where  we  overestimate  the  performance  and  underestimate  the 
cost  are  those  we  choose  to  develop  and  procure.  The  worse  the  error,  the  more 
likely  we  are  to  choose  the  system. 

The  long  and  short  run  implications  of  the  way  we  handle  R  and  D  decisions 
was  pointed  out  by  A.  W.  Marshall  and  W.  H.  Meckling  as  long  ago  as  1959  and 
by  Burton  Klein  even  earlier.  Meckling  and  Marshall  remarked  that  the  R  and 
D  decisions  made  today  affect  tomorrow's  menu  of  systems  and  that  the  decision 
as  to  how  many  we  continue  with  ought  to  be  a  function  of  our  confidence  in  the 
estimate.  The  more  confident  we  are  in  the  estimate,  the  less  we  should  duplicate, 
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because  we  can  choose  earlier.  Given  our  experience  with  the  estimates,  the 
chances  of  providing  an  inadequate  or  inappropriate  menu  are  apparent.  Finally, 
they  point  out  that  short-run  decisions  about  what  to  procure  this  year  or  next 
are  influenced  by  what  we  believe  will  be  available  2  or  3  years  from  now.  In 
ASW  forces,  this  is  a  serious  and  important  consideration. 

In  spite  ot  the  significance  of  these  issues,  it  is  not  apparent  that  there  is 
a  systematic  examination  of  these  important  allocation  questions.  The  handling 
of  R  and  D  is  not  unrelated  to  my  earlier  remarks  about  uncertainty.  We  will 
never  eliminate  it.  But  we  must  address  much  more  directly  the  question  of 
how  much  we  are  willing  to  pay  in  time  and  resources  to  reduce  it. 
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